Webtransport support quic/http 3 - #63827
Conversation
|
Review requested:
|
Building Node.jsDepending on what platform or features you need, the build process may If you can reproduce a test failure, search for it in the Table of contents
Supported platformsThis list of supported platforms is current as of the branch/release to InputNode.js relies on V8 and libuv. We adopt a subset of their supported platforms. StrategyThere are three support tiers:
Platforms may move between tiers between major release lines. The table below Platform listNode.js compilation/execution support depends on operating system, architecture, For production applications, run Node.js on supported platforms only. Node.js does not support a platform version if a vendor has expired support
Supported toolchainsDepending on the host platform, the selection of toolchains may vary.
Official binary platforms and toolchainsBinaries at https://nodejs.org/download/release/ are produced on:
Starting with Node.js 25, official Linux binaries are linked with OpenSSL asm supportOpenSSL-1.1.1 requires the following assembler version for use of asm For use of AVX-512,
AVX-512 is disabled for Skylake-X by OpenSSL-1.1.1. For use of AVX2,
Please refer to https://docs.openssl.org/1.1.1/man3/OPENSSL_ia32cap/ for details. If compiling without one of the above, use Previous versions of this documentSupported platforms and toolchains change with each major version of Node.js. Building Node.js on supported platformsPrerequisites
Unix and macOSUnix prerequisites
Installation via Linux package manager can be achieved with:
FreeBSD and OpenBSD users may also need to install macOS prerequisites
macOS users can install the Nix integrationIf you are using Nix and direnv, you can use the following to get started: echo 'use_nix --arg sharedLibDeps {} --argstr icu small' > .envrc
direnv allow .
make build-ci -j12Most dependencies will likely be available in the official nixpkgs cache, The use of ./configure $CONFIG_FLAGS
make -j12Passing the cat -> .envrc <<'EOF'
use nix --arg sharedLibDeps '{
inherit (import ./tools/nix/sharedLibDeps.nix {})
openssl
zlib
;
}'
EOFPassing the The use of # Passing `--arg loadJSBuiltinsDynamically false` to instruct the compiler to
# embed the JS core files so it is no longer affected by local changes
# (necessary for getting useful benchmark results).
# Passing `--arg devTools '[]' --arg benchmarkTools '[]'` since we don't need
# those to build node.
nix-shell \
--arg loadJSBuiltinsDynamically false \
--arg devTools '[]' --arg benchmarkTools '[]' \
--run 'make build-ci -j12'
mv out/Release/node ./node_old
# ...
# Make your local changes, and re-build node
nix-shell \
--arg loadJSBuiltinsDynamically false \
--arg devTools '[]' --arg benchmarkTools '[]' \
--run 'make build-ci -j12'
nix-shell --pure --run './node benchmark/compare.js --old ./node_old --new ./node http | Rscript benchmark/compare.R'There are additional attributes you can pass, see Building Node.jsIf the path to your build directory contains a space, the build will likely To build Node.js: ./configure
make -j4Important If you face a compilation error during this process such as We can speed up the builds by using Ninja. For more The The above requires that After building, setting up firewall rules can avoid Running the following script on macOS will add the firewall rules for the sudo ./tools/macos-firewall.shInstalling Node.jsTo install this version of Node.js into a system directory: [sudo] make installRunning testsTo verify the build: make test-onlyAt this point, you are ready to make code changes and re-run the tests. If you are running tests before submitting a pull request, use: make -j4 test
To run the linter, use To fix auto fixable JavaScript linting errors, use If you are updating tests and want to run tests in a single test file tools/test.py test/parallel/test-stream2-transform.jsYou can execute the entire suite of tests for a given subsystem tools/test.py child-processYou can also execute the tests in a test suite directory tools/test.py test/messageYou can execute tests that match a specific naming pattern using the wildcard tools/test.py test/parallel/test-stream-*
tools/test.py parallel/test-stream-* # The test/ prefix can be omitted
# In some shell environments, you may need to quote the pattern
tools/test.py "test/parallel/test-stream-*"The wildcard # Matches test/sequential/test-inspector-*, test/paFootnotes
|
|
Thanks @martenrichter, super interesting! This is great and I'd definitely be keen to include WebTransport as soon as we can, though it'll probably have to wait until nghttp3 merge the draft this depends on, which in turn looks like it's currently waiting on the final RFC. This approach in rough concept looks like it'll fit neatly with the HTTP/3 API changes & internals split I'm going to propose shortly (aiming for a PR on Monday I think) so there shouldn't be a big practical issue to implement WebTransport with that. In practice for this code specifically I should warn you there's going to be massive conflicts line-by-line since I'm moving lots of things around, but it's largely just relocation rather than more substantial changes. The only relevant that might change in approach is the ticket data and application options part, since I'm adding a new API to better split transport params from app ticket data, and pull application options entirely out from QUIC into the specific application implementation. It'll make sense shortly I hope 😄 watch this space. |
I also expected this. So I am prepared with branch to do the rebases from time to time.
Ok, I will look into it and rebase after it is merged. Most of the wt stuff is in addition to the existing mechanism, so it should not be much work.
I will see, but it looks quite orthogonal to the wt changes. |
|
"Make all old test run" (client side) done. |
7ad3e62 to
39e2915
Compare
|
Go to the newer tests and fix them (done!) (Except the flow control features not implemented in nghttp3). |
Both done. (No change no node.js required for these steps.) |
|
Cool to see this making progress. How far off are we from putting one or two small tests to demo the whole setup into here? That'd make it easier to discuss the API and try it all our end-to-end. Looks like the only tests so far are for settings config. I had a bit more of a look here after your comment in #64192, I was curious about the I think similarly to the HTTP/3 split and the existing HTTP/2 API, it would be nice to aim for a protocol-specific API that hides those internals, to expose the low-level protocol primitives but not the protocol internal working. I.e. to use the API, imo you shouldn't need to think about control streams and session ids, you just get the WT session CONNECT headers, you can allow or reject, and then you get easy to use streams within. Maybe we have a Just food for thought, we don't need to define all this immediately anyway. We can focus on the internals in the short term and work out the user-facing surface later, I'm just starting to spitball about how it's all going to glue together. |
In principle, we have this already in my external test setup. I can also run it node server and node client. (Though I do it step by step). Once I am ready with these test I will port them over.
That is because I have postponed it until my test are running. I know this is not ideal for looking into the api. But you can look here: All these files are in principle the steps to interact with the api, which would also be visible in a test.
Here are all event handlers of a wt session stream: (And if you want, I can also write, how to setup the tests, while I have not ported them to node native).
That is in principle, what my webtransport package is currently doing. Though its architecture aims at the old quiche binary addon. My plan is to make this layer thinner and then eventually, if it fits, put the remaining parts to node.js. So that it comes close to your idea. Though there are only minimal changes required to hide the id. Note, I have also parsers for the capsule protocol (alternative transport on http/3 and http/2) currently only in js (what I would be happy to port it to C/C++).
Yes something like this. But it is less dramatic.
Yes, first the internal and then the glue on top. But your vision is not so different from mine. |
fbb534d to
ba65cd6
Compare
Done. (So quic client to node.js native server). Next stop: node.js native client to node.js native server (currently it crashes). |
8b05f11 to
45c3de2
Compare
45c3de2 to
ddc7cf8
Compare
5d73904 to
7a8aa8a
Compare
Without this update the streams can stall, if the chunks are close or bigger than the window size. Signed-off-by: Marten Richter <marten.richter@freenet.de>
|
node.js native client to node.js native server (done!). (Chased several bugs for at least three weekends...) |
This PR is done to discuss early changes required for webtransport.
It is not intended for merging, as it uses an experimental PR of nghttp3, so rebases are required in the end.
Main objective is to discuss also the API on the way.
Currently, I use during the development the higher level functions of my node.js package, which has a special branch for the webtransport implementation:
https://github.com/fails-components/webtransport/tree/nodenativequic (branch for the implementation)
Relevant files are in:
https://github.com/fails-components/webtransport/tree/nodenativequic/main/lib/http3native
I have only the bare minimum of functions in node.js, as I believe that will give developers more control. So W3C interface is my package for now. (but also here will be feedback valuable)
Currently client, session and stream are partially implemented.
I am going step by step through all test I have.
I pass test 1 of:
https://github.com/fails-components/webtransport/blob/nodenativequic/main/old_test/testsuite.js
(old test, usually first to check when implementing;).
Test 2 fails. (Probably not a WT specific issue #63784).
The plan is :
But it will take time.
@pimterry @jasnell : Please invite other people who should take part in the discussion.